Process and system for generating a full color image of multispectral image from the image data of a CCD image sensor with a mosaic color filter

ABSTRACT

Process and system for producing color images and/or multispectral images in which an optical arrangement (3) images a scene on a CCD image sensor (2), said sensor being covered by a mosaic filter with a periodically repeating basic pattern (GM). A following analog/digital converter (5) produces digital signals from all pixels p ij  of the CCD image sensor (2). The basic pattern (GM) consists of a minimum of five different filter elements. In addition, an image memory (6) is provided which is divided into image memory levels (B 0 , B 1 , . . . , B nF ). The process comprises the follow steps: 
     storing of values (wert i! j! 0!) of each individual pixel (p ij ) of the CCD image sensor (2) in a dedicated level B 0  of the image memory (6), said values being determined by the CCD image sensor (2) and converted by a following analog/digital converter (5) into digital values; 
     determining an &#34;unfocussed image&#34; in the following way: for each color f and for each pixel p the neighboring pixels of color f are measured, their values read from B 0  and combined to arrive at a weighted average value; 
     storing of values (wert i! j! f!) for all the different filter elements on the CCD image sensor in dedicated image memory levels (B 1 , B 2 , . . . B nF ); 
     calculating final color values (C 1 , C 2 , . . . ,C nF ) by means of additive image sharpening; and 
     storing the final color values (C 1 , C 2 , . . . ,C nF ) in dedicated image memory levels (B 1 , B 2 , . . . B nF ).

FIELD OF THE INVENTION

The invention relates to a process for generating a full color image ormultispectral image from the image data of a CCD image sensor with amosaic color filter, said process having an optical arrangement capableof imaging a scene on a CCD image sensor which is covered by a mosaicfilter with a periodic basic pattern.

In addition, the invention relates to a system for generating a fullcolor image or a multispectral image from the image data of a CCD imagesensor with a mosaic color filter, said system having an opticalarrangement for imaging a scene on a CCD image sensor, said CCD imagesensor being covered by a mosaic filter with a periodic basic patternand being provided with an analog/digital converted for deliveringdigital signals from all pixels of the CCD image sensor.

BACKGROUND OF THE INVENTION

Electronic cameras, equipped with only one matrix image sensor (e.g., aCCD image sensor) for taking images and which nevertheless produce acolor image with a single exposure, exist both in the form of videocameras (moving picture) and electronic digital cameras (still image). Amosaic filter (CFA=color filter array) allows such cameras todifferentiate between colors. Such a filter is placed directly on thesurface of the CCD image sensor while each cell (pixel) of the CCD imagesensor is covered by a filter element (pixel filter) having suitablychosen spectral transparency.

The document "The Seybold Report on Publishing Systems"(12 Dec. 1994)discloses a digital camera which can take images with a single exposureof a CCD image sensor. The individual cells of the CCD image sensor arecovered by four different filter elements and the filter elements areplaced directly on the individual cells of the CCD image sensor. Thefour filter elements employed for taking an image are red, green, blueand blue-green. The filter elements are arranged on the CCD image sensorin a mathematical pattern. In order to determine the color of eachindividual pixel of the CCD image sensor, the color values are firstdetermined by an algorithm over an area of 64 pixels and afterwards eachpixel is compared with its nearest neighbor. If this is not sufficientto determine the color, then the comparison with the nearest neighborsis extended to a larger area. In transition regions from one color toanother, up to one thousand calculations are required for one pixel.Since the data are not available in standard RGB format, a conversionhas to be performed which takes a high performance computerapproximately 7 minutes per image to complete.

Video camera also exist which have combinations of additive andsubtractive filter sets, such as, for example, green, cyan, magenta,yellow or green, white, cyan, yellow (see EP-A-0 570 204). Suchcombinations are intended to better adjust the spatial scanningfrequencies for luminance and chrominance signals to the bandwidths ofthe video system and hence, a reduction of the color Moire effects.However, such combinations do not achieve, nor are they aimed atimproving color fidelity or improving the spectral selectivity of theimage. In the video systems discussed in EP-A-0 570 204 the signalprocessing is performed in "real time" without intermediate storage ofimages. Thus, the technical set-up is completely different than in thecase of digital still image cameras as referred to in the processaccording to the present invention.

Most electronic color cameras have mosaic filters with three differentcolors. In almost all electronic cameras these different colors arearranged in rows and columns in a periodically repeating pattern. Thepattern created by repetition of a (2,2) matrix with identical diagonalelements is known as a Bayer pattern and is widely used as a mosaiccolor filter. Highly developed processes for color interpolation inmosaic filters with RGB filters in the Bayer pattern (U.S. Pat. No.5,373,322 and U.S. Pat. No. 5,382,976) exploit the peculiarities of thispattern in a decisive way, that is, they use the dominance of green andthe low incidence of fuzziness in the case of red and blue. In addition,the processes disclosed in U.S. Pat. No. 5,373,322 and U.S. Pat. No.5,382,976 have been developed in such a way that regardless of thecontents of the image, the process is able to decide whether more useshould be made of the row-wise or of the column-wise correlations of theimage data. This produces texture-like artifacts in those image areaswhich have no details, but which are noisy.

An electronic camera has to be able to produce image data sets in aconventional format, i.e., TIFF or TARGA. This means that a value has tobe provided for each pixel and for each color belonging to the data set.Thus, for each pixel an R-value, a G-value, and a B-value have to beavailable in the RGB data set. It is evident that the raw data of thecamera do not satisfy this requirement: they contain only one colorvalue for each pixel, that is for a color which changes from pixel topixel. Hence, the camera must have a means by which it can use the rawdata to calculate these missing color interpolation of mosaic filtersor, in short, a process for color interpolation.

The intention is to arrive at values which would be obtained when takingimages without using a mosaic filter at arbitrarily short time intervalswith the different filter elements nF and using a filter wheel whichcontain the nF filter colors as filters covering the overall surfacearea.

The quality of such a process is a complicated matter. As the lack ofobject information is at best replaceable by plausible, but never bycompletely certain assumptions, it is always possible to constructinstances of images whose quality would seemingly indicate that theprocess is a failure. For example, it is easy to see that for each setof raw data there is a distribution of white light which would generatethese raw data. On the other hand, there are often plausible reasons whysuch phenomena need not be taken into consideration. The most noticeableartifacts ("aliasing") occur in the case of images in which the imagesignal differs greatly from pixel to pixel. In most electronic cameraswith relatively few pixels, a "blur filter" ensures that no such finelystructured light distributions can occur in the area of the sensor.

Distinctions are made between the following, partly overlapping qualitycomplexes:

1. Faithfulness to detail: the ability to reproduce fine structureswhich extend only over a few pixels, the reproduction being made withoutgreatly changing the form. When comparing two different systems,improved faithfulness to detail is evidenced by improved legibility ofthe text, the elements of which only cover a few pixels.

2. Artifacts: these include all image structures of which, withoutpossessing detailed knowledge of the camera, it can be said that theyare very probably not the same as the original. Examples are:

a) color edge artifacts: strongly colored striped pattern in thetransition area between evenly, but differently colored areas.

b) color Moire: semi-even, usually striped color variations overperiodically structured areas.

c) flat-field artifacts: textures, defects and jumps in intensity alongthe lines of such image areas which apparently lie outside the focalarea of the optical image and which, thus, do not show any sharpdifferences in intensity.

3. Color fidelity: considered here for monochromatic areas which extendover many pixels and in all directions.

SUMMARY OF THE INVENTION

It is the object of the present invention to avoid the undesiredappearance of texture produced by the aforementioned processes whilesimultaneously maintaining high image sharpness and suppressing coloredge defects.

A further object of the present invention is to develop and to realize aprocess in a system so that further developments are possible such as,for example, the introduction of additional filter colors, withoutnecessitating changes in the hardware and software architecture.

In addition, the process is intended for use in more periodic andpseudo-random distributions of the filters while maintaining similarlyfavorable results. In particular, the aim is to improve color fidelityand to enable a conversion of the exposure to a type of light withdifferent spectral composition.

In accordance with the present invention this object is attained bymeans of a process in which a basic pattern is built up of at least fivedifferent color filter elements and which comprises the following steps:

storing values of each pixel of the CCD image sensor in a dedicatedlevel of an image memory, said values being measured by the CCD imagesensor and being transformed by a following analog/digital converterinto digital values;

deriving an "unfocussed image" so that for each color f and for eachpixel p, the neighboring pixels of color f are determined, their valuesread from Bo and combined to yield weighted average values;

storing the values for all the different filter elements on the CCDimage sensor in dedicated image memory levels;

calculating the final color values by means of additive image sharpeningand

storing the final color values in the dedicated image memory levels.

In accordance with the present invention a system is also provided inwhich the basic pattern is built up of at least five different filterelements.

Advantages of the process and the system as disclosed by the presentinvention consist in that images can be taken by the CCD image sensorwith a single exposure and in that greater color faithfulness is therebyachieved than in the case of an RGB image. The pixels of the CCD imagesensor are covered by at least five different filter elements. Accordingto a preferred embodiment of the invention, the so-called mosaic filterconsists of a periodic basic pattern.

A particularly advantageous aspect of the present invention is theprocess implemented in the system. To generate a full color image,values of the individual pixels on the CCD image sensor are stored in anelectronic memory. Using these data the pixels are determined which areprovided with a filter element of a specific color. The color values ofthe individual pixels are determined using the corresponding weightingfactors of their nearest neighbors. The final color values aredetermined by means of additive image sharpening.

In addition, the process and its related system make it possible, whentaking an image, to convert the values from one illumination situationto another with a different spectral composition.

BRIEF DESCRIPTION OF THE DRAWINGS

The subject matter of the present invention will now be described withreference to an embodiment shown in the drawing wherein:

FIG. 1 represents a schematic view of the. system as described in thepresent invention;

FIG. 2 shows a top view of a part of the CCD image sensor in which thedistribution of the filter elements on the surface of the CCD imagesensor can be seen;

FIG. 3 depicts a top view of a part of the CCD image sensor for which adifferent distribution of the filter elements on the surface of the CCDimage sensor has been chosen, and

FIGS. 4a and 4b show a flow diagram of the process.

DETAILED DESCRIPTION OF THE INVENTION

The system according to the present invention, as shown schematically inFIG. 1, comprises an optical arrangement 3 which images a scene or anobject on a CCD image sensor 2. The CCD image sensor 2 is divided intoseveral pixels p_(ij), i running from 1 to n and indicating the numberof columns on the CCD image sensor 2, and j running from 1 to m andindicating the number of lines on the CCD image sensor 2. The signalsgained from the individual cells of the CCD image sensor 2 are conductedvia an electrical connection 4 to an analog/digital converter 5.Thereafter, the digital signals are stored in an image memory 6. Theimage memory 6 is divided into several image memory levels B₀, B₁ toB_(nF), nF representing the number of different filter elements placedon the CCD image sensor 2. The individual image memory levels B₀, B₁ toB_(nF) are connected to a processor 8 which processes the data requiredto generate a full color picture. In addition, processor 8 is connectedto an additional electronic device for the processing and storage ofsignals.

If the number of filter elements with different spectral transparency isincreased, then it is possible to achieve an improvement in colorfidelity. A CIE ruling with regard to color values for the lightincident on the CCD image sensor 2, requires that the spectralsensitivity of the pixels can be represented as a linear combination ofthe corresponding functions of the CIE standard observer. This demand isvery difficult to meet from a technical point of view and is grosslyviolated in electronic cameras of conventional construction. The resultis that no matter how well a system comprising camera and display iscalibrated, it will not permit a faithful reproduction of color nuances.To be more precise: there will always be illuminated objects whichappear to the observer as being heterochromatic for which, however, anRGB camera can only deliver like signals. Regardless of the method ofprocessing the signals, the color reproduction stays the same: this isin contrast to the observer's visual perception of the original.Naturally, the opposite effect likewise occurs: objects visuallyperceived as being isochromatic are reproduced as being heterochromatic.If a mosaic filter on a CCD image sensor contains more than threedifferent filters, then new signals can be produced by forming suitablelinear combinations of the camera signals: these new signals correspondfar more exactly to the visual perception of the original than do theRGB signals. This principle was applied to four filters in the cameramodel already referred to in the "Seybold Report on Publishing Systems",12th Edition, December 1994.

In addition, a conversion from one type of illuminating light to anothercan easily be performed. If it is to be possible to draw a reliableconclusion from the color of an illuminated body to its color whenilluminated by light having another spectral composition, than it isnecessary to know the spectral properties both of the reflected light aswell as of the illuminating light. Using suitable methods of calculation(spectral reconstruction), the spectral properties can be computed ifthe signals are known which the light yields after passing throughfilters of known spectral transmission. These computations will be morereliable, the more filter elements there are available for the analysisof the light. Just five suitable filter elements, instead of the usualthree or four, will dramatically improve the reliability of thisprocess. It can be used, for example, to take photographs under low-costhalogen light conditions and then to convert to daylight. The spectralcomposition of the illumination can be determined, without using anyother devices, by photographing a white reference body having a knownspectral reflection degree under the given illumination or, by alwaysadding this reference body to the scene to be photographed.

In the preferred embodiment of the system according to the presentinvention, five different filter elements are distributed in a specificpattern on CCD image sensor 2 (see FIG. 2).

In a preferred embodiment, for example, a system is employed comprisingof five filter elements whose maximum spectral transparency has thefollowing wavelengths (Table 1):

                  TABLE 1    ______________________________________                B =  445 nm                G' = 494 nm                G =  545 nm                G" = 570 nm                R =  595 nm,    ______________________________________

The spectral transparency of the individual filter elements B, G', G, G"and R being chosen in such a way that it is possible to achieve the mostuniform possible overlapping of the individual wavelength regions. Thewavelengths in bold-face characters denote a sensor system equivalent tothat of the human eye. The wavelengths in between are termed G' (for 495nm) and G" (for 570 nm).

The individual filter elements are arranged on CCD image sensor 2 in arepetitive pattern (see FIG. 2). This repetitive pattern is used on theentire surface of CCD image sensor 2 (see Table 2).

                  TABLE 2    ______________________________________             R     G'      B             B     R       G             G"    B       R    ______________________________________

FIG. 2 shows the arrangement of the individual filter elements on theindividual pixels p₁₁, p₁₂, . . . p_(ij) of CCD image sensor 2, irepresenting the number of rows and j the number of columns. Only partof CCD image sensor 2 is shown here since the pattern given in Table 2is continued over the entire CCD image sensor 2. Thus, for example,pixel p₁₁ of the first row and the first column is covered by filterelement R, pixel p₁₂ of the first row and the second column by filterelement G', pixel p₁₃ of the first row and the third column by filterelement B, pixel p₂₁ of the second row and the first column by filterelement B, pixel p₂₂ of the second row and the second column by filterelement R, pixel p₂₃ of the second row and the third column by filterelement G, pixel p₃₁ of the third row and the first column by filterelement G", pixel p₃₂ of the third row and the second column by filterelement B and pixel p₃₃ of the third row and the third column by filterelement R. This basic pattern GM (Table 2) is continued over the entireCCD image sensor 2: this is best illustrated in FIG. 2. Another possibleform of basic pattern GM can be seen in Table 3. FIG. 3 explicitly showsthe arrangement of the individual filter elements with respect to theindividual pixels p₁₂ . . . p_(ij) of CCD image sensor 2.

                  TABLE 3    ______________________________________             G"    G       B             B     R       G             G     G'      R    ______________________________________

The basic pattern GM as illustrated in FIG. 2 and FIG. 3 are embodimentsof multispectral mosaic filters. In addition, it is likewise possible todesign multispectral mosaic filters which contain IR-sensitive, or evenUV-sensitive, filter elements. It is expedient in this instance to takethe entire spectral range for which the "naked" CCD image sensor 2 issensitive, to break it up into transmission bands while ensuring thegreatest possible uniformity and to provide each band with a mosaic typeof filter. Such a multispectral mosaic filter camera acts as an imagingspectrometer with high spatial resolution and moderate spectralresolution.

As shown in FIG. 1, the system has a digital image memory 6 with randomaccess (RAM). The image memory 6 has image memory levels B₀, B₁ toB_(nF) which correspond to the number nF of the different filterelements used in basic pattern GM. The number of image memory levels B₀,B₁ to B_(nF) is greater by one than the number of different filterelements used to build up the basic pattern GM. In the embodimentdescribed here, exposure of CCD image sensor 2 is performed and itspixel values directly transmitted to the first level B₀ : access to thepixel values wert i! j! 0! can be made for each pixel p_(ij). Thisprovides the starting point for the above-mentioned process. In thecourse of the process, additional nF image memory levels B₁, B₂ toB_(nF) are available for the storage of data wert i! j! f!, so-called"color-values". As mentioned above, nF represents the number ofdifferent filter elements. In the embodiments described according to thepresent invention, the number of different filter elements amounts tofive. After completion of the process, the image memory levels B₁, B₂ toB_(nF) have a "full picture". In addition, the system has a processor 8which reads data from image memory 6 into its own register (not shown)in order to perform calculations with the contents of its own registerand to write the results into image memory 6.

FIG. 4a and FIG. 4b show a flow diagram of the process according to thepresent invention. A scene or image is photographed by means of the CCDimage sensor 2. In the following step, the values measured by theindividual pixels p_(ij) of the CCD image sensor 2 are converted by theanalog/digital converter 5 into digital data values wert i! j! 0!. Thesedata are stored for further use in image memory level B₀. In order toobtain a full color image, all pixels p₁₁, p₁₂ to p_(1m) have to beassigned digital values for each of the nF filter colors. This step isperformed using a double loop. The outer loop runs via colors f and theinner loop via pixels p_(ij) of the CCD image sensor 2 (see FIG. 4a).Those pixels of the CCD image sensor 2 which are provided with thefilter elements of color f and which are to be found in the neighborhoodof p_(ij) are determined. This is done by performing an evaluation ofthe neighboring pixels as well be explained in detail below. Inaddition, the nearest neighbors are given a weighting factor which willdepend on the distance to starting pixel p_(ij). These data provide thebasis for calculating the values wert i! j! f! as weighted mean values.Values wert i! j! f! are then stored for further use in the dedicatedimage memory levels B₁, B₂ to B_(nF). As already mentioned,determination of color, determination of the nearest neighbors anddetermination of the color values of all pixels are steps which arecarried out for all different filter elements on CCD image sensor 2.Using a suitable process (described below), the final color values aredetermined and likewise stored for further processing in a suitablememory.

As already mentioned, it should be noted that, in order l ate the fullimage, all color values C₁ =wert i! j! 1!, C₂ =wert i! j! 2! to C_(nF)=wert i! j! f! of each individual pixel p_(ij) on the CCD image sensor 2have to be determined for each filter color used. It can be seen fromthe preferred embodiments shown in FIG. 2 and FIG. 3, that when makingan exposure, not all pixels of the CCD image sensor 2 provide a signalfor each of the filter colors used; hence, the color values missing forcertain pixels have to be determined. The mosaic filter illustrated inFIG. 2 is taken as an example. Processor 8 reads those pixels from theimage memory level B₀ which have a value, for example, for the colorvalue R. The values for those pixels without a measured value then haveto be calculated. According to FIG. 2, a value for color value R isavailable for pixels p₁₁, p₂₂, p₃₃, p₁₄, p₂₅, p₃₆, p₁₇, . . . p₄₁, p₅₂,p₆₃, p₄₄, p₅₅ etc. Employing a suitable process, color value wert i! j!R! must be calculated for the missing pixels p₁₂, p₁₃, p₁₅, p₁₆ . . .p₂₁, p₂₃, p₂₄, p₂₆ . . . p₃₁, p₃₂, p₃₄, p₃₅ . . . p₄₂, p₄₃, p₄₅ etc. Thecalculation of the values of the other filter colors G, G', G" and B iscarried out accordingly.

Processor 8 proceeds as follows to calculate the color values wert i! j!f! for each pixel p_(ij), wherein i=1, . . . 1; j=1, . . . , m and f=1,. . . ,nF.

In a first step the following calculations are made for each color f(numbered 1,2, . . . nF) and for each pixel p_(ij).

Loading all pixel values wert i! j! 0! by using "evaluation ofneighboring pixels" (see description below) wherein the pixels belong tocolor f and whose pixels p_(ij) are nearest to pixels p_(ij), and loadthem together with their weighting factors from the image memory levelB₀. (See description of the evaluation of neighboring pixels).

Calculating the weighted mean of these data with regard to theaforementioned weighting factors W_(i'j'f).

Storing of the weighted mean as color value wert i! j! f! in the imagememory level B₁, B₂, . . . B_(nF) belonging to color f.

In this way a full, but nevertheless intermediate image is obtained andone which is termed here as being an "unfocussed image". The intentionin the construction of this image is as follows: the picture should befree of fine details and its color should be reliable in regions inwhich the color undergoes no rapid change. In order to ensure theabsence of fine details, the evaluation of neighboring pixels must beperformed in such a way that it never defines only one pixel as beingthe nearest neighbor. Only in this instance is it possible for theprocess to generate weighted mean values which are not local and whichproduce the desired reduction in sharpness. It has been shown that amean value over four or five points is ideal. If computing effort andcomputing time do not pose a problem, it is possible to make the"unfocussed picture" more homogeneous by a smoothing filter operation.

Thereafter, processor 8 calculates the final image according to aprocess for which two alternative embodiments are given here:

first, loading of the following values for each pixel p_(ij) isperformed:

S=wert i! j! 0! from image memory 6 and there from image memory level B₀;

C₁ =wert i! j! 1! from image memory level B₁ ;

C₂ =wert i! j! 2! from image memory level B₂ and

C_(nF) =wert i! j! nF! from image memory level B_(nF).

Determine color f of pixel p_(ij) according to the method of colorevaluation as described in detail in the following.

The final color values C₁, C₂, . . . C_(nF) are calculated according tothe formula (additive image sharpening)

    C.sub.k =C.sub.k +S-C.sub.f for k=1, . . . nF

or the formula (multiplicative image sharpening) ##EQU1##

If the result falls below the lowest meaningful digital signal value (asa rule 0), or exceeds the highest meaningful digital signal value(frequently 255), then these limitations are defined as representing theresult. In this instance it is preferable to reduce the additivecorrection value S-C_(f) or the multiplicative correction value S/C_(f)to a value which will prevent exceeding these limits.

The final color values C₁, C₂, . . . C_(nF) are stored in the memorylevels B₁, B₂, . . . B_(nF). They are:

wert i! j! 1!=C₁,

wert l! j! 2!=C₂. . .

wert i! j! nF!=C_(nF)

With this process it is possible to produce a barely noticeable change(using the additive method) or no change at all (using themultiplicative method) in the color of the unfocussed picture and to usethe actual signal value S to effect luminance adaptation. In the case ofthe additive method this is achieved by adding white or black, in otherwords by means of color desaturation. In this way, any color edgeartifacts become less noticeable: for this reason, the additive methodis generally preferred. In addition, it is simpler to calculate.

It is to be expected that the process will show little tendency towardcolor edge artifacts: firstly, because the color is calculated byaveraging all values, and secondly, because the color is desaturatedfollowing each addition of light. Since the brightness can be adjustedfor each pixel without averaging, good fidelity to detail is also to beexpected. Experiments have confirmed both expectations.

On the other hand, an artifact is present which is unknown in otherprocesses. Along lines where heavily colored areas meet, a colorlesspattern is generated which is caused by the geometry of parts of themosaic filter. It is generally not very noticeable.

The following gives a detailed explanation regarding color determinationand evaluation of neighboring pixels for the preferred embodiments ofthe invention.

Color determination: (determination of the color of the individualpixels p_(ij) of the CCD image sensor)

In the case of a periodic mosaic filter (basic pattern GM), asillustrated in FIG. 2 or FIG. 3, the entire mosaic filter on the CCDimage sensor 2 is built up by periodic repetition of basic pattern GM.The column number of the basic pattern is termed nVert and the rownumber mHor. Thus, nVert=mHor=3 is valid for the pentaspectral patternproposed in accordance with the present invention. The distribution ofthe colors on the nver, mHor positions of basic pattern GM takes theform of a numerical (nVert, MHor) matrix. Basic pattern GM, for example,has the form shown in Table 3 in the case of the pentaspectral mosaicfilter (for the sake of clarification the filter elements were arrangedas indicated in FIG. 3):

                  TABLE 3    ______________________________________             4     3       1             1     5       3             3     2       5    ______________________________________

The numbers in the matrix are assigned as follows to the individualcolors of the filter elements of the basic pattern: B≈1, G'≈2, G≈3, G"≈4and R≈5.

On the basis of the matrix shown in Table 3, that pixel p_(ij) isdetermined which has a specific color f=C_(ij). Color f of pixel p_(ij)is calculated by means of the following equations in which i is thevertical and j is the horizontal position of an arbitrary element of theCD image sensor and in which the row-wise and column-wise position ofthe pixel on the basic pattern GM is represented by i_(lokal) andi_(lokal) respectively.

i_(lokal) =1+remainder of the division (i-1)/nVert

j_(lokal) =1+remainder of the division (j-1)/nHor

C_(ij) =GM i_(lokal) ! i_(lokal!)

In a pseudo-random mosaic filter, color C_(ij) of pixel p_(ij) isdetermined for a given i and j as follows:

r:=50000·sin (i)+30000·sin (j)

z1=biggest integer, smaller or equal to r.

r1=r-z1, this number yields 0<r1<1.

In order to illustrate the significance of this expression, we consider

y: 5·sin (i)+3·sin (j) as a decimal fraction and assume, for the sake ofsimplicity, that y is positive:

y=d1, d2 3 d4 d5 d6 d7 d8

The number r1 can then be developed into a decimal fraction

0, d6 d7 d8

r1 is defined by the "higher digits" of y; it is evident that theydepend on i and j pseudo-randomly. In a last step, r1 is adjusted to thedesired value range (1, . . . nF). (nF is the number of the differentcolors):

r2=nF·r1

z2=biggest integer, smaller or equal to r2

C_(ij) =1+r2-z2.

This is only one of many procedures which are based on the fact that thehigher decimal digital of transcendental functions dependpseudo-randomly on the argument of the function, if it varies in bigsteps compared to the order to magnitude which is given by the decimaldigit in question. In particular, the constants 50000 and 30000 can bereplaced by other numbers of a similar order of magnitude, and it ispossible to replace sin () by cos (). This purely random distribution ofcolors has the tendency to generate clusters, that is pixel regionsoften occur in which one color is far more frequently represented thanin the case of even distribution. There are likewise surprisingly largeareas in which one color is completely missing. By varying the process,as can be done in processes for "digital half-toning", care can be takenthat the color distribution is more uniform, but without giving rise tolocally constant patterns, such as those which are present in periodicmosaic patterns. An obvious method is to take optimizing experiments asa basis and to develop such "uniformized random distribution" of thepixel colors over a relatively large field of about 64·64 pixels andthen to perform periodic repetition of the pseudo-random pattern. As aresult, the advantages of periodic patterns, i.e. the simpler neighboralgorithm, are also effective in this case.

The evaluation of the neighboring pixels

(determination of the number of pixels p_(ij) located closest to aparticular pixel which have the same color as the first pixel).

As not all pixels p_(ij) of the CCD image sensor 2 supply theinformation of a simple color, the missing pixels p_(ij) have to becalculated in order to obtain a color image. The process proposed formaking these calculations contains several parameters to be evaluatedwhich can be freely chosen within certain limits. First of all, thedistance between the pixels is defined. The distance d of pixel p_(i1j1)to pixel p_(i2j2) is defined as follows:

    d(i1,j1;i2,j2):=|i1-i2|+|j1-j2|

or, the geometrical distance between the pixel P_(i1j1) and pixelp_(i2j2), the latter, however, being less suitable for fast computing.

In addition, for each pixel p_(ij) a calculation has to be made todetermine the maximum area U(p_(ij)) to be taken into consideration;this fixes an upper limit for the number of "neighboring pixels" anddefines the number of computing steps required to select theseneighboring pixels. This is performed by establishing a number d_(Max)with which it is possible to calculate the area U(p_(ij)) in question.

    U(p.sub.ij)={p.sub.i'j' ||i-i'|≦d.sub.Max ; |j-j'|≦d.sub.Max ; C.sub.ij =f}

For the color occurring the least frequently, the set should also haveat least 2 and preferably 4 to 6 elements. This can be achieved byselecting a sufficiently large value of d_(Max). This means, however,that for the color occurring the most frequently, the set U(p_(ij),;f)can contain more pixels than are required. In order to take this intoconsideration, the pixels are arranged according to their distance (seeabove) from the reference pixel p_(ij) and those furthest away arediscarded.

The following procedure is applied: two pixels from this set are definedas being equivalent when they have the same distance from referencepixel p_(ij). In this way, U(p_(ij) ;f) is divided into equivalenceclasses. Arranged according to their increasing distance, theseequivalence classes are denoted as follows:

U₁ (p_(ij) ;f); U₂ (p_(ij) ;f); . . . ; U_(k) (p_(ij) ;f)

and the number of their elements are denoted by

u₁ (p_(ij) ;f), u₂ (p_(ij) ;f), . . . u_(k) (p_(ij) ;f).

The above-mentioned possibility of discarding some of these elements ismade reality by selecting a natural number k0 with

1≦k0≦k

such that ##EQU2## According to the definition NU(p_(ij) ;f)=U₁ (p_(ij);f)∪U₂ (p_(ij) ;f)∪. . . ∪U_(k0) (p_(ij) ;f)

then yields pixels p_(ij) of color f which are in the area closest topixel p_(ij). The weight of pixels p_(ij) now has to be evaluated. Onepossibility is that this weight solely depends on the distance to thereference pixel; thus, all pixels of the same equivalence class have thesame weight. This is done by choosing a list ω of k0 positive numbersfor which the following is valid:

ω(k0)≦. . . ≦ω(1) ##EQU3## Hence the weight of a pixel P_(ij) εNU(p_(ij) ;f) is given by

    w(p.sub.ij ; f):=ω(p)/U.sub.p (p.sub.ij ;f)

where the index p is chosen so that

(p_(ij))=U_(p) (p_(ij) ;f)

Thus, the weighted mean value required in the first step of the processis defined as ##EQU4##

In this way, the evaluation of the neighboring pixels can be performedeach time a color is determined. However, the formation of equivalenceclasses is very computer intensive. In the case of a periodic pattern,computer time can be significantly reduced by making preliminarycalculations and storing them. In equation (B1)i and j outside thesquare brackets can be replaced by i_(lokal) and j_(lokal), defined inthe earlier description of the color evaluation, and i', j' by i+s, j+twhere (s,t) is restricted by corresponding conditions like (i',j'):##EQU5## with NU_(rel) (p_(ij) ;f):={(s,t)|(i+s,j+t) ε NU(i,j;f)}

With this, for the nVert·nHor·nF values (i_(lokal), j_(lokal) ;f), thelists

NU_(reL) (p_(ilokal) jlokal ;f)=((s1,t1),(s2, t2) . . . ) and

w(p_(ilokal) jlokal ;f)=(w₁,w₂, . . . )

have to be calculated, whose length is typically (see above) only 4 to6. These lists are calculated once as part of the software developmentof the system and stored in a ROM. When this process takes place in acamera with the actual values (p_(ij) ;f) being dependent on(p_(ilokal), jlokal ;f), the relevant lists are loaded and the weightedmean value simply calculated as:

    wert i! j! f!=w1·wert i+s1! j+t1! 0!+w2·wert i+s2! j+t2!0!+. . .

The present invention has been described with reference to a preferredembodiment; however, it is obvious that an expert in the field may makemodifications in accordance with his capabilities without exceeding thescope of protection of the following claims.

I claim:
 1. Process for generating color images and/or multispectralimages in which an optical arrangement (3) images a scene on a CCD imagesensor (2), said sensor being covered by a mosaic filter with a periodicbasic pattern (GM), characterized in that the basic pattern (GM)consists of five different color filter elements and in that the processcomprises the following steps:storing values (wert 1! j! 0!) of eachindividual pixel (p_(ij)) of the CCD image sensor (2) in a dedicatedlevel B₀ of an image memory (6), said values being measured by the CCDimage sensor (2) and being transformed by a following analog/digitalconverter (5) into digital values; determining an "unfocussed image" bycalculating a neighboring pixel of color f for each color f and for eachpixel p, reading their values from B₀ and combining them to arrive at aweighted average value; storing values (wert i! j! f!) for all thedifferent filter elements on the CCD image sensor (2) in dedicated imagememory levels (B₁, B₂, . . . , B_(nF)); calculating of final colorvalues (C₁, C₂, . . . , C_(nF)) by means of additive image sharpening;and storing the final color values (C₁, C₂, . . . , C_(nF)) in dedicatedimage memory levels (B₁, B₂, . . . , B_(nF)).
 2. Process according toclaim 1, characterized in that the final color values (C₁, C₂, . . . ,C_(nF)) are calculated by multiplicative image sharpening.
 3. Processaccording to claim 1, characterized in that illumination prevailing whenphotographing a scene can be transformed into an illumination of adifferent spectral composition.
 4. Process according to claim 3,characterized in that when photographing a scene, a white reference bodywith known spectral reflection properties is added to the scene to bephotographed.
 5. Process according to claim 3, characterized in that adata memory of processor (8) stores the data for differentilluminations, said data being transferable from the memory according toa chosen illumination, or characterized in that the processor (8) usesdata obtained from an illuminated reference body to calculate the datanecessary to make a change to a different spectral illumination andstores them in the memory of processor (8).
 6. Process according toclaim 1, characterized in that the basic pattern (GM) used in the mosaicconsists of a pentaspectral pattern.
 7. Process according to claim 6,characterized in that a distance from a starting pixel is defined insuch a way that in an area defined by a chosen distance there are atleast two pixels with the least frequently occurring color f of a filterelement.
 8. Process according to claim 7, characterized in that in theneighborhood of the starting pixel there are four to six pixels with theleast frequently occurring filter color.
 9. Process according to claim6, characterized in that the pixels at equal distance from a startingpixel are assigned to an equivalence class and in that these pixels havea same weighting factor w.
 10. Process according to claim 9,characterized in that equivalence classes and the weighting factors forthe periodic basic pattern (GM) are stored in the memory of processor(8) in order to reduce a time to produce a final color image.
 11. Systemfor generating color images and/or multispectral images by means of anoptical arrangement (3) which images a scene on a CCD image sensor (2),said CCD image sensor (2) being covered by a mosaic filter with aperiodic basic pattern (GM) and connected to a following analog/digitalconverter (5) which produces digital signals from all pixels p_(ij) ofthe CCD image sensor (2), characterized in that the basic pattern (GM)is built up of at least five different filter elements, and a digitalimage memory (6) with random access is provided, said memory having anumber of image memory levels (B₁, B₂, . . . B_(nF)) which correspondsequally to the number of different filter elements of the basic pattern(GM) and having in addition an image memory level B₀ which stores thedigital signals of all pixels p_(ij) of the CCD image sensor (2), saiddigital signals being generated by the analog/digital converter (5). 12.System according to claim 11, characterized in that a processor (8) isused to control the system, organization and access of a digital imagememory (6) and to evaluate digital image data.
 13. System according toclaim 11, characterized in that the basic pattern (GM) is apentaspectral pattern.
 14. System according to claim 11, characterizedin that each individual level of image memory levels (B1, B2, . . .B_(nF)) stores only data of a specific color value wert i! j! f!, and inthat color values wert i! j! f! consist of the digital data of the colorvalues of pixels p_(ij) of the CCD image sensor (2), said pixels p_(ij)being provided with filter elements of the specific color f, and in thatthe digital values for pixels p_(ij) of the CCD image sensor (2) whichare not provided with the filter element of the specific color f can beevaluated with the help of the processor (8).
 15. System according toclaim 14, characterized in that a distance from a starting pixel isdefined in such a way that in an area defined by a chosen distance thereare at least two pixels of the least frequently occurring color of afilter element.
 16. System according to claim 15, characterized in thatin the neighborhood of the starting pixel there are four to six pixelswith the least frequently occurring filter color.
 17. System accordingto claim 16, characterized in that those pixels equidistant from thestarting pixel are assigned to an equivalence class and in that thesepixels have a constant weighting factor.
 18. System according to claim17, characterized in that in the case of a periodic basic pattern (GM)the equivalence classes and the weighting factors are stored in thememory of processor (8).
 19. System for generating color images and/ormultispectral images by means of an optical arrangement (3) which imagesa scene on a CCD image sensor (2), said CCD image sensor (2) beingcovered by a mosaic filter with a periodic basic pattern (GM) andconnected to a following analog/digital converter (5) which producesdigital signals from all pixels pij of the CCD image sensor (2),characterized in that the basic pattern (GM) is built up of at leastfive different filter elements, and a pentaspectral pattern is built upof three pixel rows and three pixel columns and in that pixels p₁₁, p₂₂and p₃₃ are provided with a red filter element R, pixels p₁₃, p₂₁ andp₃₂ with a blue filter element B, pixel p₂₃ with a green filter elementG, pixel p₁₂ with a green filter element G' and pixel p₃₁ with a greenfilter element G", the maximum spectral transparency of G' being lessthan the maximum spectral transparency of G".
 20. System according toclaim 19, characterized in that the maximum spectral transparency of redfilter element R is 595 nm, the maximum spectral transparency of bluefilter element B is 445 nm, the maximum spectral transparency of greenfilter element G is 545 nm, the maximum spectral transparency of greenfilter element G' is 494 nm and the maximum spectral transparency ofgreen filter element G" is 570 nm.
 21. System for generating colorimages and/or multispectral images by means of an optical arrangement(3) which images a scene on a CCD image sensor (2), said CCD imagesensor (2) being covered by a mosaic filter with a periodic basicpattern (GM) and connected to a following analog/digital converter (5)which produces digital signals from all pixels pij of the CCD imagesensor (2), characterized in that the basic pattern (GM) is built up ofat least five different filter elements, and a pentaspectral Bayerpattern is built up of three pixel rows and three pixel columns, and inthat pixels p₂₂ and p₃₃ are provided with a red filter element R, pixelsp₂₁ and p₁₃ with a blue filter element B, pixels p₁₂ and p₂₃ with agreen filter element G, pixel p₃₂ with a green filter element G' andpixel p₁₁ with a green filter element G", the maximum spectraltransparency of G' being less than the maximum spectral transparency ofG".